{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Keras 高级用法及其他\n",
    "\n",
    "## Keras Function API\n",
    "\n",
    "在 Keras 中使用 `Sequential` 定义的模型，只能有一个输入和一个输出，这常常是不够灵活的。要实现更多样的网络结构，需要用到 Function API。\n",
    "\n",
    "下面两个代码块定义了两个模型，分别采用 `Sequential` 和 Function API 写法，但这两个模型本质上都是一样的。在 `Sequential` 这个类中，只需要添加各种功能的层，该类能够帮助完成数据的传递，本质就是把数据以流水线的形式输入给一个一个的层得到最终结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_4 (Dense)              (None, 32)                2080      \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 32)                1056      \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 3,466\n",
      "Trainable params: 3,466\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential, Model\n",
    "from keras import layers\n",
    "from keras import Input\n",
    "\n",
    "seq_model = Sequential()\n",
    "seq_model.add(layers.Dense(32, activation='relu', input_shape=(64,)))\n",
    "seq_model.add(layers.Dense(32, activation='relu'))\n",
    "seq_model.add(layers.Dense(10, activation='softmax'))\n",
    "seq_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用 Function API 则由自己编程来完成数据的流动，因此更加灵活。在使用 `Sequential` 时，把 `layers.Dense(32, activation='relu')` 当做一个层，其实 `layers.Dense(32, activation='relu')`  本质上是一个函数，它对输入做运行后得到输出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 32)                2080      \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 32)                1056      \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 3,466\n",
      "Trainable params: 3,466\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "input_tensor = Input(shape=(64,))\n",
    "x = layers.Dense(32, activation='relu')(input_tensor)\n",
    "x = layers.Dense(32, activation='relu')(x)\n",
    "output_tensor = layers.Dense(10, activation='softmax')(x)\n",
    "\n",
    "model = Model(input_tensor, output_tensor)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 有多个输入的模型\n",
    "\n",
    "![](https://wangyu-name.oss-cn-hangzhou.aliyuncs.com/superbed/2019/05/11/5cd6a5753a213b04174bc8e1.jpg)\n",
    "\n",
    "针对上面这种模型，下面就故意造一个例子，对 IMDB 的影评进行分类，两个输入分别是评论文本和翻转的评论文本。本质可能就和双向 LSTM 差不多吧。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "vocab_size = 10000\n",
    "max_len = 200\n",
    "\n",
    "(input_train, y_train), (input_test, y_test) = imdb.load_data(num_words=vocab_size)\n",
    "\n",
    "# pad_sequences 的功能是让所有序列都一样长，长度截断，短的补零\n",
    "input_train = sequence.pad_sequences(input_train, maxlen=max_len)\n",
    "input_test = sequence.pad_sequences(input_test, maxlen=max_len)\n",
    "\n",
    "# 把样本中的所有句子倒转过来\n",
    "input_train_reversed = input_train[:,::-1]\n",
    "input_test_reversed = input_test[:,::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import layers, Input, Model\n",
    "\n",
    "text1_input = Input(shape=(None,), dtype='int32', name='text1')\n",
    "embeded_text1 = layers.Embedding(vocab_size, 128)(text1_input)\n",
    "coded_text1 = layers.LSTM(32)(embeded_text1)\n",
    "\n",
    "text2_input = Input(shape=(None,), dtype='int32', name='text2')\n",
    "embeded_text2 = layers.Embedding(vocab_size, 128)(text2_input)\n",
    "coded_text2 = layers.LSTM(32)(embeded_text2)\n",
    "\n",
    "concatenated = layers.concatenate([coded_text1, coded_text2], axis=-1)\n",
    "\n",
    "y = layers.Dense(50, activation='relu')(concatenated)\n",
    "label = layers.Dense(1, activation='sigmoid')(y)\n",
    "\n",
    "model = Model([text1_input, text2_input], label)\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 47s 2ms/step - loss: 0.4494 - acc: 0.7794 - val_loss: 0.4239 - val_acc: 0.8080\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.2715 - acc: 0.8899 - val_loss: 0.5014 - val_acc: 0.7892\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.2124 - acc: 0.9185 - val_loss: 0.3269 - val_acc: 0.8790\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.1769 - acc: 0.9343 - val_loss: 0.3242 - val_acc: 0.8710\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.1360 - acc: 0.9503 - val_loss: 0.3815 - val_acc: 0.8776\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.1049 - acc: 0.9619 - val_loss: 0.3558 - val_acc: 0.8710\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.0807 - acc: 0.9718 - val_loss: 0.5125 - val_acc: 0.8596\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.0661 - acc: 0.9759 - val_loss: 0.5379 - val_acc: 0.8706\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.0472 - acc: 0.9841 - val_loss: 0.5618 - val_acc: 0.8604\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 44s 2ms/step - loss: 0.0393 - acc: 0.9870 - val_loss: 0.6413 - val_acc: 0.8576\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f80380f50f0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([input_train, input_train_reversed], y_train,\n",
    "          epochs=10, batch_size=128, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看样子还真能 work，不过因为参数较多，产生了 overfitting。\n",
    "\n",
    "前面 `model.fit` 函数还可以像下面这样调用，即使用一个 map 传入参数。这里 map 的 key 就是前面定义 Input 时给定的 name。\n",
    "\n",
    "```python\n",
    "model.fit({'text1': input_train, 'text2': input_train_reversed},\n",
    "          y_train, epochs=10, batch_size=128, validation_split=0.2)\n",
    "```\n",
    "\n",
    "## 层共享\n",
    "\n",
    "上例中，Embedding 层就没有必要出现两个，两个输入可以经过同一个 Embedding 层，然后分别经过不同的 LSTM 层。如此，就需要对 Embedding 层进行共享。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 49s 2ms/step - loss: 0.4457 - acc: 0.7869 - val_loss: 0.3474 - val_acc: 0.8630\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 45s 2ms/step - loss: 0.2694 - acc: 0.8950 - val_loss: 0.5314 - val_acc: 0.8056\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 46s 2ms/step - loss: 0.2151 - acc: 0.9181 - val_loss: 0.3093 - val_acc: 0.8682\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 46s 2ms/step - loss: 0.1862 - acc: 0.9303 - val_loss: 0.3333 - val_acc: 0.8682\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 45s 2ms/step - loss: 0.1537 - acc: 0.9435 - val_loss: 0.4061 - val_acc: 0.8754\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f7ca1ab7ef0>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras import layers, Input, Model\n",
    "\n",
    "embedding = layers.Embedding(vocab_size, 128)\n",
    "\n",
    "text1_input = Input(shape=(None,), dtype='int32', name=' ')\n",
    "embeded_text1 = embedding(text1_input)\n",
    "coded_text1 = layers.LSTM(32)(embeded_text1)\n",
    "\n",
    "text2_input = Input(shape=(None,), dtype='int32', name='text2')\n",
    "embeded_text2 = embedding(text2_input)\n",
    "coded_text2 = layers.LSTM(32)(embeded_text2)\n",
    "\n",
    "concatenated = layers.concatenate([coded_text1, coded_text2], axis=-1)\n",
    "\n",
    "y = layers.Dense(50, activation='relu')(concatenated)\n",
    "label = layers.Dense(1, activation='sigmoid')(y)\n",
    "\n",
    "model = Model([text1_input, text2_input], label)\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['acc'])\n",
    "\n",
    "model.fit([input_train, input_train_reversed], y_train, epochs=5, batch_size=128, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Keras Callback\n",
    "\n",
    "Callback 就是在训练的各个特定的点调用一些函数，这样用户可以干预到训练，比如早停，或者显示训练进度、打印日志、保存模型等等。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import keras\n",
    "\n",
    "class EpochTimeLogger(keras.callbacks.Callback):\n",
    "    def __init__(self):\n",
    "        self.epoch_start_time = None\n",
    "    def set_model(self, model):\n",
    "        self.model = model\n",
    "    \n",
    "    def on_epoch_begin(self, epoch, logs=None):\n",
    "        self.epoch_start_time = time.time()\n",
    "        self.epochs = self.params['epochs']\n",
    "    \n",
    "    def on_epoch_end(self, epoch, logs=None):\n",
    "        now = time.time()\n",
    "        duration = now - self.epoch_start_time\n",
    "        print('Epoch %d/%d - %.1fs' % (epoch + 1, self.epochs, duration))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3 - 24.7s\n",
      "Epoch 2/3 - 24.5s\n",
      "Epoch 3/3 - 24.5s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f80858345c0>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "callbacks_list = [\n",
    "    EpochTimeLogger()\n",
    "]\n",
    "\n",
    "model.fit([input_train, input_train_reversed], y_train,\n",
    "          callbacks=callbacks_list, verbose=0,\n",
    "          epochs=3, batch_size=512, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorBoard\n",
    "\n",
    "在终端输入 `tensorboard --logdir=logs` 后，打开浏览器在 6006 即可看到。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/5\n",
      "20000/20000 [==============================] - 25s 1ms/step - loss: 0.0407 - acc: 0.9909 - val_loss: 0.5636 - val_acc: 0.8572\n",
      "Epoch 2/5\n",
      "20000/20000 [==============================] - 25s 1ms/step - loss: 0.0493 - acc: 0.9884 - val_loss: 0.5545 - val_acc: 0.8634\n",
      "Epoch 3/5\n",
      "20000/20000 [==============================] - 25s 1ms/step - loss: 0.0100 - acc: 0.9975 - val_loss: 0.5821 - val_acc: 0.8140\n",
      "Epoch 4/5\n",
      "20000/20000 [==============================] - 25s 1ms/step - loss: 0.0325 - acc: 0.9915 - val_loss: 0.6092 - val_acc: 0.8682\n",
      "Epoch 5/5\n",
      "20000/20000 [==============================] - 25s 1ms/step - loss: 0.0298 - acc: 0.9911 - val_loss: 0.6682 - val_acc: 0.8682\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f7afb3c1048>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "\n",
    "callbacks = [\n",
    "    keras.callbacks.TensorBoard(\n",
    "        # Log files will be written at this location\n",
    "        log_dir='logs',\n",
    "        # We will record activation histograms every 1 epoch\n",
    "        histogram_freq=1,\n",
    "    )\n",
    "]\n",
    "\n",
    "model.fit([input_train, input_train_reversed], y_train,\n",
    "          callbacks=callbacks,\n",
    "          epochs=5, batch_size=512, validation_split=0.2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
